Quantum impurity in the bulk of topological insulator 
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We investigate physical properties of an Anderson impurity embedded in the bulk of a topological 
insulator. The slave-boson mean-field approximation is used to account for the strong electron 
correlation at the impurity. Different from the results of a quantum impurity on the surface of a 
topological insulator, we find for the band-inverted case, a Kondo resonant peak and in-gap bound 
states can be produced simultaneously. However, only one type of them appears for the normal 
case. It is shown that the mixed-valence regime is much broader in the band-inverted case, while 
it shrinks to a very narrow regime in the normal case. Furthermore, a self-screening of the Kondo 
effect may appear when the interaction between the bound-state spin and impurity spin is taken 
into account. 

PACS numbers: 71.27.+a, 73.20.Hb, 75.20.Hr 



I. INTRODUCTION 

Topological insulator (TI) is a novel state of quantum 
matter, which is insulating in the bulk, but hosts con- 
ducting edge or surface states near the system boundary. 
It has attracted attention in the community of condensed 
matter physics due to its potential application in spin- 
tronics and quantum computation* 1 - - — A class of materi- 
als such as Bi2Se3 and Bi2Te3 have been found to pos- 
sess the surface states which form a single Dirac cone4^— 
Suppression of backscattering inside the Dirac cone guar- 
antees that the Dirac dispersion remains essentially un- 
perturbed for weak perturbation that preserves time- 
reversal symmetryi 7 -^ So far, the effects of the impurity 
scattering on the surface of TIs have been investigated 
extensively^ - — 2 In the presence of classical spins which 
break time-reversal symmetry, it was predicted that the 
impurity could open up a local gap and suppress the lo- 
cal density of states The suppression of backscat- 
tering around nonmagnetic impurities on the surfaces 
with strong spin-orbit coupling has been confirmed by 
scanning tunneling miscroscope experiments J^r— How- 
ever, strong nonmagnetic scattering, such as electrostatic 
potentials, may disrupt the Dirac cone and create low- 
energy impurity resonances^ For a quantum impurity 
on the surface of TIs, the Hamiltonian for the impurity 
can be mapped to the conventional pseudo-gap Anderson 
model. The impurity is fully screened at low temperat- 
ues when the Fermi level is located away from the Dirac 
point J&i 7 - 

Although there are theoretical and experimental stud- 
ies on the quasi-particle states around an impurity, most 
of which focus on the impurity on the surface of TIs. Es- 
sentially, the topological nature of TIs is determined by 
the electronic structure of the bulk bands instead of the 
surface states. On the other hand, the TI samples avail- 
able nowadays are always poorly insulating in the bulk, 
owing to a large mount of vacancies and defects 2£t2lL For 
these reasons, it is important to study how the quasi- 



particle states are affected when vacancies or impurities 
are localized in the bulk of the system. It has been 
shown that classical spinal or vacancie o 19 ' 20 localized in 
the bulk of TIs could result in the coexistence of in-gap 
bound states and the boundary states. For a quantum 
impurity, the quantum fluctuations of its internal degree 
of freedom play an important role, making them signif- 
icantly differ from classical impurities. 28 However, it re- 
mains unknown how the quantum impurity in the bulk 
of TIs affects the electronic states. 

Actually, the study on the quantum impurity in the TIs 
is also related to the problems of the impurities in the un- 



conventional density waves 



28-31 



gapped systems j ^ or 



spin-orbit-coupled systems ! 34 ' 35 For instance, in a gapped 
system, the Kondo effect breaks down when the energy 
gap exceeds a critical valued In the presence of Rashba 
spin-orbit interaction, a parity-breaking Dzyaloshinsky- 
Moriya term could be induced, resulting an exponential 
enhancement of the Kondo temperature^ Since there 
are strong spin-orbit coupling and unconventional gap in 
the TIs, it becomes interesting to investigate the differ- 
ences between the Kondo effects in conventional insula- 
tors and TIs. 

In this paper, we investigate the effects of a quantum 
impurity embedded in the TIs with the help of the slave- 
boson mean-field approach. We show that in-gap bound 
states and Kondo effect could coexist in TIs, while only 
one type of them appears for conventioal insulators. If 
the bound states are singly occupied, the Kondo reso- 
nance could be screened by the exchange interaction be- 
tween the impurity spin and the spin of the impurity- 
induced bound states, leading to a self-screened Kondo 
effect. The paper is organized as follows. In Sec. II we 
introduce the model Hamiltonian of the impurity in TIs 
and the slave-boson mean field approach. In Sec. Ill we 
discuss the Kondo effect and the formation of the in-gap 
bound states in both band-inverted and normal cases. In 
Sec. IV, we show the self-screening of the Kondo effect. 
Finally, a summary is presented in Sec. V. 
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II. MODEL HAMILTONIAN AND SLAVE 
BOSON APPROACH 



occupancy on the impurity is allowed. We introduce the 
auxiliary fields 



A. Model 

The effective model to describe the bulk states of TIs 
with an impurity is written as 



H — Hn + Hd 



(1) 



The part for the bulk electrons of TI is given by the 
modified Dirac model 36-38 



H = ^\[hv F k- 



a 



Bh 2 k 2 ) 



1** 



with cti = a x ® Oi and (3 = a z ® uq , where Oi (i = x, y, z) 
are the Pauli matrices and ctq is the 2x2 unit ma- 
trix, ki = —idi (i = x, y, z) is the momentum opera- 
tor, k 2 = k 2 + k 2 + k 2 , vf and m have the dimensions 
of speed and mass, respectively. Different from the sur- 
face Hamiltonian, a quadratic correction in momentum 
—Bh 2 k 2 and a gap term mv 2 F are introduced in the bulk 
Hamiltonian. The sign of mB determines whether the 
system is topologically trivial or not: it is non-trivial for 
mB > (i.e., the band-inverted case), and trivial for 
mB < (i.e., the normal case) i^£r— The energy spec- 
tra have a finite energy gap. Here the basis vectors are 
chosen as 



*£ = [ a lv a U> b lv b l 



where at and bt are creation operators of electrons 
with spin a on two different orbits. In this representa- 
tion, we can rewrite the total Hamiltonian in the second- 
quantized form 

H = y^ y (mv F - Bh 2 k 2 )(al a a kcr - b\ a b klJ ) 



ka 



k 



+(k x - ik y )(al t bki + b[ t a k i) + h.c.]. (2) 
The Hamiltonian that describes the Anderson impurity 



4b, 



where the boson operator W creates an empty state and 
the fermion operator d\ creates a singly occupied state 
on the impurity. These two fields obey the local con- 
straint b'b + ^2i a cftd = I^ In the mean- field approxima- 
tion, both the annihilation and creation boson operators 
b and are replaced by a complex number bo and its 
complex conjugate &q, and the local constraint is realized 
by introducing a Lagrangian multiplier Aq. Substituting 
them to the original Hamiltonian ([3]) and ((H), one can 
get 

H d = i d {d\d t + d\di.) + Xo(b 2 - I) 



and 



H = ^(V^ajLA + Vbkb\ a d a 



h.c.) 



ka 



with the renormalized parameters = + Aq and 



v, 



a(b)k 



b V a 



(b)k- 



The slave-boson mean-field approxi- 



mation was firstly introduced to describe the low-energy 
physics of the conventional Anderson impurity model 
in the mixed-valence regime,^ This method may pro- 
duce the low-energy physics in unconventional density- 
wave (e.g., d-wave superconductors^ graphene electron 
systems r 2 - etc.). 



C. Green's functions 

Utilizing the method of the equation of motion for 
the impurity electron, one finally obtains the retarded 
Green's function of the impurity electron 



«<kl4» 



l d - SoM + erE z (w) 



[a,-e d -£oM] 2 -£i [EiMr 
where the self-energy functions are defined as 



(5) 



H d = ed(4 t c dt + c di Cdi) + Uc dt Cdtc di c d i, (3) 

where e d is the impurity energy level and U is the onsite 
Coulomb interaction. The coupling Hamiltonian between 
the impurity and electrons in TI has the form 

Ht = J2( Vaka L c d<r + Vbkbl a c d a + h.c), (4) 

ka 

where V a ^ k represents the hybridization strength. 
B. Slave-boson mean field 



Ej(w) 



(w + A k )V 2 k + (w - A k )V b 



i,k 



k 



uj 2 — h 2 Vpk 2 

2flVFkiVakVbk 



h 2 Vpk 2 



(6) 



(i = x, y, z), and A k = mv 2 F — Bh 2 k 2 . 

It is assumed that the hybridization strength V a ^)k 
does not depend on momentum, V a (b) k — V a n,). In the 
absence of external magnetic field, the interaction self- 
energy = and the electronic Green's function 
can be simplified as 



Here we consider a strong on-site Coulomb interaction 
on the impurity, i.e., U — > oo. In this limit, no double 



(7) 
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which has the same form as those of the conventional 
Anderson impurity model. Additionally, we consider the 
case that the impurity is symmetrically coupled with two 
orbits, i.e., Vo = V a = Vb, and the self-energy E (w) in 
(i-dimension [d — 2, 3) is thus simplified as 



E (w) 



•E 



2V 2 



uj 2 - h 2 v 2 F k 2 - A 2 



l.d 
hp 



dk 



k d ~ 



uj 2 — h 2 vpk 2 — A 2 , 



(8) 



where No is the number of lattice sites and kp is the 
Fermi wave vector for the bulk of TIs. 



D. Normal and band-inverted regimes 

From the poles of self-energy function Eo(w), one ob- 
tains the energy spectrum of the bulk 



i{k) = ±^Jh 2 v 2 F k 2 + (mv\ - Bh 2 k 2 ) 2 . 

The energy spectrum shows a complex dependence on 
the energy gap and momentum. Such energy spectra 
can be mapped onto many important cases of impu- 
rity problems. Near the band edges, the model re- 
duces to the Anderson problem in normal insulators 
or semiconductors! 32 ' 33 The quantum impurity model in 
graphene or surface states of the TIs can be recovered by 
setting m = and B = O^L 22 ^ 2 

It can be seen from the energy spectrum that for mB < 
1/2, the band edges are located at k — and the energy 
gap is 

A N = 2\m\vp 

as in normal insulators. However, for the case of mB > 
1/2, the band edges appear at a finite k with a gap 

\J\mB - 1 , 



For convenience, one defines 

dN V 2 e 
.90 (e) = — 



Ve 2 -A 2 /4- 



l-2mB „2 
2\B\ U F 



4(\B\h 2 k 2 F )i^e 2 - A 2 /A 
when e > An/2, and 



(9) 



Si(e) 



dN V 2 e\^/e 2 -A 2 /A + 



2mB-l „,2 
2\B\ F 



A{\B\h 2 k 2 F )i^/e 2 - A 2 /A 



(10) 



when both A//2 < e < Ajv/2 and mB > 1/2 are satis- 
fied. 



For the case of mB < 1/2, one gets 

1 1 



E (w) 



UJ - £ UJ + £ 



(11) 



After analytical continuation, Eo(oj) = ReEo(w) 
ImEo(w) with 



POO 

ReEoM = P / d€go (£) 



_w - £ uj + £ 



(12) 



and 



A 



ImEoH = -g (uj)Q(\u>\ - (13) 



with P denoting the principal value. 

For the case of mB > 1/2, the real and imaginary parts 
of self energy are given by 



ReEoH = P 



oj — £ oj + £ 



(14) 



and 



ImEo (a;) 



52 ( w )e(M-^)e(^-M) 



+ flb ( w )e(H-^) 



(15) 



where 52 (w) = 50 (w) + 31 (w). Because ImEo is propor- 
tional to the density of states of the bulk electrons, the 
density of states is discontinuous at this point An/2. For 
a relatively large B and small m, the energy gap reduces 
to a value much smaller than A n ■ 

Minimizing the free energy of the system with respect 
to Aq and bo, one obtains a set of self-consistent equations 



dw/(w)p d (w) + 6g - 1 = 0, 



2/ dw/(w)(w-e d )p d (w) + A 6o = 0, (16) 

J — 00 

where /(w) = [l+e^ _A1 )/ fcsT ]" 1 is the Fermi distribution 
function, p is the chemical potential of the TI, and the 
density of states Pd{oj) of the impurity is given by 



Pd{u) = - 



ImEo (oj ) 



7T (oj - i d - ReE (w)) 2 + ImEoH 2 ' 



In the calculation, it is limited to zero temperature, 
i.e., ksT = 0. An /2 is taken as the energy unit and Po = 
ttpqVq represents the hybridization strength between the 
impurity and the bulk electrons, where po = No/ 2D is 
the density of states per spin at the chemical potential 
and D = ^ (hvFkp) 2 + (mvp — Bh 2 kp) 2 is a cut-off of 
the band width. In the following, we show the results of 
quantum impurity in three-dimensional TIs for T = 0.5 
and D = 30.0. 
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III. IN-GAP BOUND STATES AND THE 
KONDO EFFECT 

A. Self-energy 

First, we discuss the self-energy Eo(o>), which depends 
on the energy spectrum of the bulk bands. In the bulk 



of the TI, the strong spin-orbit coupling couples the con- 
duction and valence bands, leading to the non-parabolic 
energy spectrum. The self energy So(w) reveals not only 
the poles of the Green's function, but also the inhomo- 
geneous density of states due to the impurity-induced 
states. 




FIG. 1: The real and imaginary parts of the self-energy ~Eq(lj) for different mB values. 



Figure [T] shows the real and imaginary parts of So(^) 
versus lo for different values of mB. Previous studies 
based on the Chern number and Z2 invariant indicate 
that the band-inverted case with mB > is topologi- 
cally nontrivial while the normal case with mB < is 
topologically trivial.— ~— It is shown that in Fig. Q] both 
the real and imaginary parts of So(w) are quite different 
for opposite signs of mB. The real part ReSo(^) deter- 
mines the level positions of both the Kondo resonance 
and the bound states, according to the solutions to the 
equation 

w = e rf + Re£ M- (17) 

For mB > 1/2, ReSo(w) diverges rapidly near the edges 
of the gap, thus there always exists the bound states 
within the gap. When the impurity level | | is much 
larger than A//2, the bound states are very closed to 
the bottom of the conduction bands for ^ A//2 or 



the top of the valence band for <C — A//2, which is 
similar to the case of s-wave superconductors^ More 
importantly, Eq. (|17[) has an extra solution in the band 
region, corresponding to the coexisting Kondo resonance. 
For mB < 0, there is only one solution to Eq. flTTl) . so 
the Kondo effect and the bound states do not appear 
simultaneously. 

The imaginary part ImSo(w) is proportional to the 
density of states of bulk electrons. For mB > 1/2, the 
energy gap is A/, thus ImSo( w ) = in the region of 
w £ [-A//2, A//2]. At the band edges ±A//2, ImS (w) 
shows divergences. In the region \lu\ £ [Aj/2, Ajv/2], the 
divergences drops rapidly with the increasing \uj\. For 
\w\ > Aat/2, ImEo(w) begins to increase as a function 
of y/ui. For < mB < 1/2, the divergences at the band 
edges ±Ajv/2 disappear gradually with decreasing mB. 
For mB < 0, ImI]o(a;) at the band edges is always zero 
and increases as a function of ^/uj for \uj\ > Ajv/2. 
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FIG. 2: The density of states of the impurity electron are shown for different impurity levels td = —2.0, —1.0, —0.5, 0.5, 1.0, 
and 2.0 (from left to right) in (a) band-inverted case mB = 1.0 and (b) normal case mB — —1.0, where the chemical potential 
fj, = 0.0 lies in the gap. 




FIG. 3: The density of states of the impurity electron are shown for different impurity levels e c i = £dc (fdc is the critical 
point at which e<j = fi), —2.5, —0.5, 0.5, and 2.0 (from left to right) in (a) band-inverted case mB = 1.0 and (b) normal case 
mB = —1.0, where the chemical potential /i = —2.0 lies in the valence band. 



B. Density of states of the impurity 

From the self-energy, the Kondo resonance and the 
bound states have been discussed qualitatively. The den- 
sity of states /3d(w) of the impurity for the cases that the 



chemical potential fi lies in the gap and in the valence 
bands are presented in Figs. [5] and [3J respectively. It 
is demonstrated that in both cases the Kondo resonance 
and bound states coexist for mB > 0, while only one 
of them appears for mB < 0. For mB > 0, the bound 
states are very close to the band edges when Id is far 
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FIG. 4: The positions of the bound states and low-energy 
Kondo resonance peak as functions of impurity level in (a) 
the band-inverted case mB = 1.0, 0.4 and (b) the normal case 
mB = —1.0, —0.4. The chemical potential is taken as /i = 0.0. 



away from the energy gap. In the previous studies^ - — 
it has been argued that when the energy gap exceeded 
an critical value (A/Tr- = 2.0 predicted by slave-boson 
mean- field theory 3 -^ and Tk is the Kondo temperature), 
the Kondo effect does not appear any more in the in- 
sulators with an energy-independent density of states. 
However, for a complex dispersion relation, the density 
of states is strongly energy-resolved and quite different 
near the band edges for the band-inverted and normal 
cases. In the band-inverted case, /9rf(±A//2) equals zero 
exactly, due to the divergence of ImEo(w) near the band 
edges. While for \ui\ ^> A/, pd{u>) also approaches to 
zero. Therefore, no matter how \x lies in the gap or in 
the bands, there are always two low-energy resonance 
peaks lying near the band edges for mB > 1/2, as shown 
in the upper panels of Figs. [5]and|31 When the chemical 
potential fi lies in the valence band, the low-energy reso- 
nance peak becomes narrow and its position moves to /i 



with decreasing e^. Near the critical point tdc the reso- 
nance peak becomes very sharp and close to the chemical 
potential. 

In Fig. 4 we present the positions of the in-gap bound 
states and low-energy Kondo resonance as functions of 
the impurity level in the normal and band-inverted 
cases. The bound states show quite different behaviors 
in the two cases. For the normal case, the position of the 
bound states is almost linear in e^. The in-gap bound 
states start from the point tdc = — Ajv/2, corresponding 
to the top of the valence bands. At the bottom of the 
conduction bands, the bound states enter the conduc- 
tion bands continuously, and then the low-energy Kondo 
resonance peak appears. However, for the band-inverted 
case, the bound states in the gap and the Kondo res- 
onance can form simultaneously for the impurity level 
(d > tdc- In this case, the positions of these two states 
do not connect at any point. For an ed far away from the 
chemical potential, the in-gap bound states are very close 
to the band edges. Thus they are vulnerable to small per- 
turbation or thermal fluctuation, and may merge into the 
conduction bands easily. 



Broadened mixed-valence regime for 
band-inverted case 



The mean field b\ introduced in Sec. Ill Bl gives the 
probability that the impurity is empty. Fig. [5] presents 
the dependence of b\ on the impurity energy level for 
different values of mB. b$ = when the impurity level ed 
is much lower than the chemical potential, which means 
that the impurity is singly occupied. In this case, the 
charge fluctuation between the impurity and the bulk 
bands is suppressed. When td exceeds a threshold value, 
&o begins to increase from and saturates at 1 when 
€d /i. The mix-valance regime is defined as where 6q 
changes from to 1. Fig. [5] presents b 2 when the chemical 
potential lies in the gap and in the valence bands, respec- 
tively. As shown in Fig. [SJ the mixed-valence regime is 
broader for the band-inverted case with mB > 0. For 
the normal case with mB < 0, fr 2 , increases more rapidly, 
and the mixed- valence regime shrinks into a very narrow 
regime. This kind of narrow mixed- valence regime has 
also been found in the unconventional density waves and 
d-wave superconductors! 28 ' 31 For mB < 0, the density 
of states of bulk electrons vanishes at the band edges. 
The reduction of the density of states at the band edges 
implies that the Kondo resonance peak is narrower and 
&o decreases to zero faster than those in the conventional 
case. From the self-consistent equations, near the critical 
point, the critical value of td is determined by 



tdc — 2 



^ to.*™ 



where g(co) is proportional to the density of states of the 
background electrons. In the band-inverted case, g{oj) 
diverges near the band edges, which results in a large 
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integral value as well as large |e<j c |. In contrast, g{uS) 
reduces to zero near the band edges in the normal case. 
In this case, |e<j c | is smaller and the mixed- valence regime 
becomes narrower. 




at the point ui — Ajv/2, the self-energy S (w) is finite 
in three dimensions. While in two dimensions, So( w ) 
has logarithmic divergence to — (+)oo at u> — > +(— )|m| 
for TIs and has logarithmic divergence to +(— )oo at 
uj — > — (+)\m\ for normal insulators 4^ This means when 
the Anderson impurity couples only to one band, the 
topological phase transition can be seen from the posi- 
tion of impurity bound states as the system changes from 
normal insulator to TI. Besides, the physical properties 
of the quantum impurity are similar qualitatively in two 
and three dimensions, e.g., the mixed- valence regime in 
both cases is much broader for mB > than the case of 
mB < 0. 



IV. 



SELF-SCREENING OF THE KONDO 
EFFECT 



(b) 





impurity 



bound state - 



J, 



impurity 





topological^ nontrivial case 



topological^ trivial case 



FIG. 5: The dependence of the order parameter 6q on the 
impurity energy level td for (a) the chemical potential in the 
gap jj, = 0.0 and (b) in the valence band fi — —2.0. 



D. In two dimensions 

The preceding sections present the numerical results in 
three dimensions. In this section, we briefly discuss the 
behaviors of the quantum impurity in the bulk of two- 
dimensional TIs. In practice, the two-dimensional case is 
more accessible by the scanning tunneling miscroscope.— 
At the mean-field level, the difference between the two 
and three dimensions arises from the self-energy Y*q(uj), 
which in two dimensions can be expressed as 

A detailed discussion about the properties of 12q(uj) in 
two dimensions is presented in Ref. 44, in which a 5- 
impurity scattering is considered. It can be deduced that 



FIG. 6: Illustration of the Kondo effect for (a) topological 
nontrivial case and (b) trivial case. 

We have shown that the quantum impurity may in- 
duce in-gap bound states in the band inverted case, lead- 
ing to the coexistence of the Kondo resonance and the 
bound states. Now we discuss the effects if the exchange 
interaction between the impurity spin and the impurity- 
induced bound-state spin is taken into account. When 
the chemical potential of the bulk TI lies in the conduc- 
tion bands, both the bound states and the impurity level 
are occupied, and each of them behaviors like a single 
localized spin (as illustrated in Fig. [6]). The impurity 
spin can interact with the bound-state spin, due to the 
second-order virtual hopping or the RKKY mechanism 
mediated by the conduction electrons. If the interaction 
is antiferromagnetic, the two spins form a singlet, which 
will compete with the many-body singlet formed by the 
impurity spin and conduction electrons, quenching their 
Kondo effect. Different from the intensively-studied two- 
impurity Kondo problem ) 45 ' 46 here the quenched Kondo 
effect is induced by the spin of the bound states rendered 
by the impurity itself. Therefore, we refer to this effect 
as the self-screening of the Kondo effect. 
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A. Exchange interaction between impurity spin 
and impurity-induced bound-state spin 

The self-screened Kondo effect can be illustrated by 
the model Hamiltonian as follows 

a a 

+ J2( Vdbda i° d ° + V d b dbl a d a + h.c.) 

her 

+ T.^f^-f- + V f b f b U° + h.c.) 

ha 

+\ d (Y, 4d« + b\ - 1) + A/(^ flfa + b)- 1), 

(18) 

where e/ is the impurity-induced bound-state level and 
H is the Hamiltonian for the bulk of TI (Eq. 2). The 
term JS d ■ S / describes the exchange interaction between 
the impurity spin and bound-state spin. 

B. Order parameter for the exchange interaction 



Similar to the treatment in Sec. Ill Bt two slave-boson 
operators b d and bf are introduced to replace c d (j)a by 
b d ^d(f) a in the large-U limit. The spin exchange term 
JS d ■ S f = JJ2a,a> d\d a >fl,f a can be decoupled by in- 
troducing a valence-bond field A = — In the 
mean-field approximation^ 

JS d ■ S f -+ JA V(4/ CT + fid.) + J A 2 , 



the Hamiltonian becomes quadratic in the fermion opera- 
tors. The problem is still far from trivial as b d , bf, X d , Xf, 
Ao, and e/ need to be determined self-consistcntly. Dif- 
ferent from the ordinary Kondo problem, here the bound- 
state energy ef also enters the self-consistent equations. 
By minimizing the ground-state energy, one obtains a set 
of nonlinear self-consistent equations 



bl-l 



b}-l 



(J 

^((4/ CT ) + (/trf CT )) + 2 A 

a 

J2 < y^ a L d <r) + Vd(b{ a da) + h.C.) + 2b d \ d 

52(VM a U) + V f {b\j a ) + h.c.) + 26/ A/ 



0. 
0. 

0, 
0, 

0.(19) 



A'.IT 



To simplify the calculation, we assume that the bound 
states are always singly occupied (bf — 0) and decoupled 
from the conduction electrons. In this case, the in-gap 



bound states act like a single spin. Correspondingly, the 
constraints for the bound states becomes A/(^ (T (/ ( |/ (T ) — 
1) and Xf is contained in the related Green's functions. 



C. Green's functions 

Performing the equation of motion procedure, we can 
obtain the Green's functions for the impurity and the 
bound states 



(RI4» = 



i 



(if Ma)) 



1 



Eo(w) 



-I 



.PA?, 



(20) 



w-e d -Eo(u) 

€f + Xf, and E (w) 



where i d = e d + X d , if 

2V 2 

u 2^k uj 2 -h 2 v 2 F k 2 -A 2 k ■ 

In the limit A — » 0, the impurity and the bound states 
are decoupled and the results reduce to those when J = 0. 
When the spin exchange interaction exceeds a critical J c , 
a nonzero order parameter Ao appears and the Kondo 
peak near the chemical potential is expected to split. 
From the Green's functions ((d a \dl)), one obtains the 
self-consistent equation for the bound states 



£/ - e d 



72 A 2 

° S ( e/ ) = 0. 



Defining a(uj) = [ui — i d — ReEo(w)](w — if) — J 2 A 2 , and 
/3(uj) — ImEo(a;)(aj — if), the self-consistent equations 
are derived as 



a(w) 2 +/3H 5 



b\ 



1 



" ^ J 2 A 2 Im£ H 



du 



a(w) 2 +/?H 2 
JA 0( 8(w) 



= 0, 
= 1, 



aH 2 +/3(w) 2 



A = 0, 



" d j(L,-i d )(u-i f )- j 2 Ain^) 



a(uj) 2 + 



= ^db 2 d , 



a(e/) = 0. (21) 
This set of equations can be solved numerically. 

D. Self-screened Kondo effect 

Fig. 7 presents the effects of bound-state spin on the 
Kondo effect for different exchange interaction strength 
J. The chemical potential of the bulk of TI is tuned into 
the conduction bands, so there is a kondo peak near the 
chemical potential and the bound states are occupied by 
a single electron. When J exceeds a critical value J c , the 
order parameter A begins to increase from zero, then 
quickly to 1 with the increase of J. Figure 7(b) shows the 
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J 



density of states of the impurity as a function of energy 
for J = 0.0, 0.04, and 0.08. With the increase of J, 
the resonance peak splits. The splitting of Kondo peak 
increases with J. As a result, the density of states near 
id reduces to a very small value, corresponding to the 
self-screening of the Kondo resonance. 

Above we assumed that the bound states are singly- 
occupied in the large-U limit. If the Coulomb repulsion 
energy on the bound states is finite and the chemical po- 
tential is high enough, it is possible that the bound states 
are occupied by two electrons, and they form a singlet due 
to the Pauli exclusion principle. In this limit, the bound- 
state spin is decoupled from the impurity spin, and the 
Kondo effect originated from the interaction between the 
impurity spin and conduction electrons can recover. 
V. SUMMARY 
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FIG. 7: (a) The order parameter Ao as a function of J. Pa- 
rameters: €d — 1.0, /i = 2.0, and mB = 1.0. (b) The density 
of states of the impurity for J = 0.0, 0.04, and 0.08. 



In summary, we have studied the Kondo effect and the 
formation of in-gap bound states induced by an Ander- 
son impurity coupled with the bulk states of topological 
insulators. It is demonstrated that the positions of the 
Kondo peak and bound states strongly depend on the 
topological properties, the chemical potential, and other 
parameters of the system. The behaviors of the resonance 
level in the bulk of TIs differ from those for simple met- 
als and normal insulators. Due to the divergence of the 
density of states near the band edges, the mixed- valence 
regime is much broader in the band-inverted case, while 
it shrinks to a very narrow regime in the normal case. 
For the band-inverted case, the in-gap bound states and 
the Kondo resonance can coexist. However, only one of 
them exists in the normal insulators. When the impu- 
rity energy level is far away from the chemical potential, 
the in-gap bound states are very close to the band edges 
and can be considered as merging into the bulk. Further- 
more, we show that a self-screening Kondo effect can be 
induced by taking the interaction between the impurity 
spin and bound-state spin into account. 
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